{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "### Requests methods\n",
    "\n",
    "import requests\n",
    "from requests_oauthlib import OAuth1\n",
    "\n",
    "r = requests.get('https://api.sampleurl.org/get',auth=auth)  \n",
    "r = requests.post('http://sampleurl.org/post', data = {'key':'value'})  \n",
    "r = requests.put('http://sampleurl.org/put', data = {'key':'value'})  \n",
    "r = requests.delete('http://sampleurl.org/delete')  \n",
    "r = requests.head('http://sampleurl.org/get')  \n",
    "r = requests.options('http://sampleurl.org/get')  \n",
    "\n",
    "\n",
    "r.text  \n",
    "r.json()  \n",
    "r.econding()  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Twitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#### Define urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Response [200]>\n",
      "Fidget for America 🇺🇸😜 #babefortrump #trump #America #maga #trumprally #spinner #FidgetSpinner https://t.co/SxAg5F4pZT\n",
      "RT @AndieCase: Kimpossible\n",
      "RT @CommonWhiteGirI: OMG LOOK AT THOSE LITTLE PAWS 😭😍 https://t.co/t8FUcek434\n",
      "RT @FlirtingIy: if my parents buy you food, let you stay in my house, or give you a ride... I BETTER HEAR A THANK YOU EVERY. SINGLE. TIME.\n",
      "RT @apenastii: i wish i looked like a snack but instead i look like i ate all the snacks https://t.co/du7lsH7vUe\n",
      "相互フォローします。\n",
      "『頑張るしかないです。\n",
      "だって、来年もこの学校で戦車道をやりたいから。みんなと。』\n",
      "RT @itsMarioSelman: no one got hurt we're okay😌\n",
      "So I saved myself. ☺️☺️☺️☺️\n",
      "https://t.co/78RA28dfk4  -   Past Life Regression Pack\n",
      "RT @laviejamabel: no doy https://t.co/KYFqiMHBYn\n",
      "RT @horikoshiko: このあと17:30からアニメです！先週の次回予告面白かった。 https://t.co/ixhorqfbaW\n",
      "@Shamooka_ At all Lebron got it so easy up until the ECF\n",
      "RT @CAGrizBeat: JaMychal’s agent, Michael Hodges: \"I’m looking at two offer sheets &amp; sign-and-trades. Seems to us Memphis is going in a dif…\n",
      "RT @Hania_zarad: https://t.co/O5pl9Z9XL0\n",
      "RT @jhuliecostanza: i have a lot of feelings for their relationship ok https://t.co/RaWMCKNibR\n",
      "RT @lipkitari: adorable omg https://t.co/fQ6yExLwGx\n",
      "RT @imjayhay: i want a girl with a short skirt and a https://t.co/nzzSnmRTrk\n",
      "RT @jcmonahan: Thanks Keith! https://t.co/qCqfbTUchB\n",
      "#قروب_ليال_للدعم\n",
      "#الباشق_للدعم\n",
      "#امنيه \n",
      "كل إنسان سيكون يوماً ما مجرد ذكرى\n",
      "أمنيتي أن أكون ذكرى بيضاء في قلوبكم ✨💌 https://t.co/xXFqtBAyrD\n",
      "RT @IEM: From the entire #IEM team, we wish you a happy and safe 4th of July! \n",
      "\n",
      "See you in Oakland later this year! https://t.co/XVeAS89TgB\n",
      "RT @Trav_CeeJay: Checkmate Nintendo https://t.co/kr4ic1n4Kr\n",
      "Nope. https://t.co/J5yTYBK6DU\n",
      "https://t.co/52Lf2C3sJk\n",
      "奴は災厄の存在…世界を脅かしている…　その名は…リュウザキ…アイツを…野放しにはしておけない！もう残された人類で倒すしかない！\n",
      "Y'all outta line @DaviBeckham24x7 kissed HIS 5yr old just cause y'all toddlers out here FCKN Don't stop others from ❤️\n",
      "RT @whiteQuill: اللهم وفق أميرنا تميم بن حمد لما تحبه\n",
      "و ترضاه ، وأحفظ اللهم دولتنا بحفظك\n",
      "وألهم حكام الخليج كافة البصيرة\n",
      "والحكمة لما فيه خير…\n",
      "اللَّهُمَّ إِنِّي أَعوذُ بِرِضاكَ مِنْ سَخَطِكَ، وَبِمعافاتِكَ مِنْ عُقوبَتِكَ،  https://t.co/hn1IsJxJvW\n",
      "渡辺彩香がシミュレーションゴルフ対決に「難しい」。（。・＿・。）ﾉ\n",
      "RT @Fre6teen_: https://t.co/pi1dSnz0Bw\n",
      "RT @MOVEFORWARDHUGE: THEN GET THE FUCK OUT.  AMERICA CHOSE TRUMP. WHY?  Because you lawless children, acted like idiots for 8 years and… \n",
      "RT @davis1988will: Trump, Putin to reportedly hold bilateral meeting in Germany\n",
      "https://t.co/NnjuU4JlZu https://t.co/miQ1tcVuk6\n",
      "sneezy defined: https://t.co/1rQwzuw2gz\n",
      " https://t.co/5HkWLxkMrO\n",
      "RT @Ericthomasbtc: Be in Beast Mode during the process !! This helps you #StayReady !!\n",
      "\n",
      "https://t.co/o2YRPvfaA7 https://t.co/O5YDpekWeU\n",
      "@CatalunyaRadio @CatVespre @CatMigdia @maticatradio Pero si parte de insultar a la mitad de los Cat o los ESP a Eur… https://t.co/C8FzsoDn4z\n",
      "@CEROPILOTOS joe te quiero mucho\n",
      "Forecast for Pinal County Housing, AZ: Jul 4 Moderate (Yellow), Jul 5 Moderate (Yellow)\n",
      "This is MY quote. Rude. UN RT. https://t.co/tOLpzXjn4g\n",
      "RT @c_sureyadizesi: Beklemek uzun bir sızıdır;Ömrümüze sığdırdığımız..Mehmed Uzun\n",
      "@feed_boss @ObvTheWorst stfu spencer\n",
      "@officialpepe @Besiktas Nerdesin be bittik gelicen diye 👌👌👌\n",
      "RT @imraansiddiqi: A Muslim brother was murdered in a bout of road rage, teaching his son to drive. https://t.co/AA8RdEfzms \n",
      "Help his… \n",
      "RT @Simone_Biles: watched 47 meters down then got in a cage  🦈 https://t.co/XQy5d93SR2\n",
      "RT @baetrissss: I'ma need yall to stop making it seem like cheating is a norm and we should tolerate it https://t.co/yhoo0qnsaP\n",
      "لعيون #خبيره\n",
      "ناطرتك يا #كويتى على الرابط\n",
      "https://t.co/EUAQEkMsar\n",
      "كلبه محرومه\n",
      "\n",
      "#نيك https://t.co/B1CMuLjgvE\n",
      "شكل نومهم ماتعدل هذي صلاة الفجر أمس\n",
      "😂😂 🙊 #المدينة_المنورة #الفجر https://t.co/8kg8yO1WVo\n",
      "RT @juan_phelipe8: Botei um pagodinho no som aqui e estou arrumando a casa ks\n",
      "RT @GarrettZoukis: The real ones will hold you down, not back.\n",
      "あなたはHola@ashokkapur14が苦手なフレンズなんだねー！大丈夫！フレンズによって得意なことがちがうから！\n",
      "@phoenixanew I totally had a nightmare about a picture I saw from the new Twin Peaks. Just a picture!\n",
      "@14Okmg わからない😭😭😭😭\n",
      "@GibasDev @CHellerDickinsn @washingtonpost @PostOpinions Oh.  Do they publish when these things are about to happen?\n",
      "#데이터 #대리 #댈찍 #대리찍사 합니다\n",
      "\n",
      "팬싸, 팬미팅, 지방행사, 공항 및 급한대리 등 모두 가능합니다.\n",
      "\n",
      "paypal도 가능합니다. 모든 문의는 DM 또는 오픈 카톡으로 주세요.\n",
      "\n",
      "오픈 카톡 https://t.co/cLyBfNsydH\n",
      "@gtkuwait خبول\n",
      "はぁ痛くて痛くてたまらない\n",
      "RT @ZellnerLaw: TH sub-key: not master. Hood latch swab: never swabbed  hood latch. Bullet; w/wood never shot TH. #MakingAMurderer  https:/…\n",
      "過去最大級のキャンペーン展開中！　登録してサービス利用すれば、１，０００円分のポイントがもらえる。　他にもポイント交換率アップなど！　この機会にお小遣いもらっちゃおう！　登録と詳細はこちらから！過去最大級のキャンペーン！　登録してサービス利用すれば、１，０００円分のポイントがもら\n",
      "RT @stylesdrunk_: SELEÇÃO BRASILEIRA NÉ MORES\n",
      "#LigaMundial\n",
      "RT @CloydRivers: To the Troops, police officers, EMTs, firefighters, nurses, and doctors: Thank you for working the holiday so others can e…\n",
      "RT @xtiandella_: BAM!!! It's Finally time to  #GainWithXtianDela!!\n",
      "\n",
      "#RETWEET if you are ONLINE!\n",
      "\n",
      "Let me help you gain 1,000 Followers!\n",
      "https://t.co/iVoN2tn8FO\n",
      "RT @Ramboff_713: @olkovaleva1 @SdibqJgth Соглашусь, Алёха – именно #УГ )))\n",
      "&lt;  https://t.co/9qRDusxDme\n",
      "@wajdi14003 @taraneem2016 @samar_qs @4_tnah @brave_heart_55 @loaiabid @hejazeah1 @Rose25Rosa @tufyomar @_thr_thr… https://t.co/xVNmBoJo8N\n",
      "RT @MaryaOmid: کشتن ادما راحته؟!!!!!\n",
      "چرا میکشیم!!! احساس استیصال میکنم!!! چقدر فاصله گرفتیم از انسانیت!!!! \n",
      "#FreeTaheri\n",
      "#SaveAli\n",
      "#SaveArash\n",
      "اللهم إني أعوذ بك من عذاب القبر وأعوذ بك من فتنة المسيح  ♻️  https://t.co/3a3W6hQgdL\n",
      "RT @richardsonisbae: Moja babcia (perf niemiecki, urodziła się tam) ciągle mi mówi, że źle wymawiam Andreas, bo to ma być \"Andryjas, a nie…\n",
      "RT @CassidyBaldwin: So this person has the same number as my mom and I was trying to send my mom a picture of my arm and well 😂 https://t.c…\n",
      "RT @PoetaImpostor: Tan poco tiempo de conocernos y he cumplido más sueños de los que tuve toda mi vida.\n",
      "RT @WaladShami: 8 Muslims were shot last night outside a mosque in France...where y'all French flag filters @ now?\n",
      "الذكر عند الخروج من المنزل:بسم الله توكَّلْتُ على الله https://t.co/PmuGVnYo9X\n",
      "すぴ～ https://t.co/JvHg5oUzOZ\n",
      "RT @D1V4S: alô, Ines Brasil? Oi amore, queria te convidar pra gravar um clipe... https://t.co/kcfqGAPxvF\n",
      "RT @ScottSmithTN: Happy 4th! 🎆🎇🎆🎇🎆\n",
      "Proud to be an #American 🇺🇸, where at least I know I'm free👊\n",
      "\n",
      "Today we honor the sacrifice of so m… \n",
      "Publiqué una nueva foto en Facebook https://t.co/Zg2t6otsRn\n",
      "@matthealy the cutest gals\n",
      "Somos expertos en muebles de oficina. Visítenos en la Cra 5 # 34 - 83 Montería - #MuyBuenosDias #FelizMartes… https://t.co/kDsfXf2r86\n",
      "Want vulgar, young man !  I learn here   https://t.co/dnnikp3LRJ https://t.co/gTDIwFh0yj\n",
      "الناعمه من كثر ما تعيش بـ ترف\n",
      "مـدري تـشـم الـورد ولا يـشـمها \n",
      "\n",
      "اغـمـز لها وتقـولـي يا منحرف\n",
      "وشلون اجـل لأمن بغيت اض… https://t.co/0XghbqfpWG\n",
      "@your_sutener Идеально 🌌💫 https://t.co/0LVbFwl3go\n",
      "We are people of the earth, but Scripture says that we are also people of the unseen world. - @pastormarktobey @MyFaithRadio.\n",
      "RT @garyfoskett: \"Top BBC figures in running to head Theresa May's communications\"\n",
      "\n",
      "Ever wondered who \"edits\" BBC politics progs?… \n",
      "@legitarmandoo anytime anytime (:\n",
      "RT @mhmd350: رآضي بـ آلأقدآر حتى لوّ تفآرقني\n",
      " آلحظ مآله علآقه في حجم حبّك\n",
      "\n",
      " بقول لك و آنت كيفك لآ تصدقني\n",
      "أخلصت لك لكن : آلتوفيق من ربّك..\n",
      "RT @IndirectasRock: Y sí, te extraño, pero la vida es así y hay que seguir.\n",
      "@thisnorthernboy The amazing @juliedillon as well\n",
      "Disaster , europa league cannot be won without the rangers in it, sad night for football\n",
      "RT @Babbs_com: Quando faço cenas para os outros, eu não preciso que retribuam apenas que valorizem\n",
      "RT @fb_sayfasi: Seneye Patates All stara gitsin \n",
      "Serhatta onu yorumlasın \n",
      "GönlümüzdeAdın SerhatAkın\n",
      "@_MicaCapria Pareces copada, nunca hablamos\n",
      "RT @simplykris__: ok camilla is jarring now why are u getting so defensive this guy likes u stop puttin ur barriers up #LoveIsland\n",
      "@Jablonic Tveksamt ändå va mtp det ekonomiska läget?\n",
      "&lt;  https://t.co/kddgpNaBFg\n",
      "RT @Trumpster4Trump: @PeriwinkleDsgns @Mo_An2016 @BasedMonitored Got to get all those flies sucka. https://t.co/7se9FZYTvy\n",
      "RT @johannamaska: I can't imagine - having worked for a President - how anyone in the White House has time for this crap. Get to work. http…\n",
      "#Migrantes celebran independencia de #EU con regalo https://t.co/7L2fBPRWJb\n",
      "@surenashakori البته من قطعا اون بهشت و هم نمیخوام\n",
      "RT @BreitbartNews: I did it my way. https://t.co/inObSEXHG1\n",
      "JONAH – Wicked Fever Tour https://t.co/7MKuvnnNSb\n",
      "voz da emily e muito engraçada no áudio 😂😂😂\n",
      "Or another Rabits member. Get a whole squad\n",
      "RT @Wonderlist: Hey #Toronto!\n",
      "You could WIN tickets to see Iron Maiden!\n",
      "https://t.co/MwbXqvwF9u https://t.co/Qt955DiJc3\n",
      "Felicida #VegasBaby https://t.co/Tn3Z5xUmld\n",
      "@cosmicgate @Amnesia_Ibiza hahaha, see y'all next thursday!\n",
      "「アトリエミツコ」が「スリップ【ポリエステル100％】ノンレース　スリップ　3サイズ・3色展開在庫限りで廃番になります」を発売!! #ランジェリー　#インナー\n",
      "https://t.co/HlD50XfSDz https://t.co/jPzEQqa5Hz\n",
      "Room 1 temp: 26.23\n",
      "Room 2 temp: 26.24\n",
      "Power room temp: 26.91\n",
      "Garr room temp: 26.23\n",
      "Trafo room temp: 15.66\n",
      "Ext temp: 28.13\n",
      "@LeidianeMello6 @whindersson eu sei kkkkkkk\n",
      "RT @LetrasUm44k: A gente conversava todo dia, lembra?\n",
      "\" أصعب ما في النقاش ليس الدفاع عن وجهة نظرك، بل معرفتها. \" أندريه موروا\n",
      " https://t.co/03v9iOGdv8\n",
      "RT @AboSananh511: محمد رسول الله \n",
      "صلى الله عليه وسلم https://t.co/anGJLZJvjx\n",
      "https://t.co/ctNAowH3eN  -   Wiredfigs.\n",
      "RT @aziz_1973: ولي العهد السعودي💚🇸🇦💚السماح ب #قيادة_المرأة_للسيارة يسعد اكثرمن10مليون مواطنه وسيسجل التاريخ اسمك بحروف من ذهب وسيد… \n",
      "RT @saintlagerfeld: YOUNG ARIANA RUN POP. https://t.co/GayXi3qokx\n",
      "Buena e importante prueba d fuego para nuestros jugadores. Enfrentamientos d esta índole,son interesantes,para mejo… https://t.co/wYN2jcXjMg\n",
      "con mi corazon te espero roberto ledezma canta  arbey https://t.co/Lq3v4ewgo5 vía @YouTube\n",
      "The A-Z of Networking- E is for... https://t.co/jTONsoChTd #TwitterTips #motivation https://t.co/tjgS0adtlG\n",
      "#كفو_يا_قطر https://t.co/vKfMxuYt1R\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RT @fluffy_pnk: wir sind an einem Punkt, wo Kekse, Isomatten, Zwiebeln und Schlafsäcke die kapitalistische Weltordnung gefährden! #fastgesc…\n",
      "قحبه جامعيه مع زميلها يركبها دوجى ويصورها..https://t.co/v0f9Pcq7TX\n",
      "\n",
      "سحاقيات\n",
      "روابط_سكس\n",
      "hot\n",
      "LOZV https://t.co/2vjN2pAiNa\n",
      "@MarianKeyes Ah your a hoot I rest my suitcase\n",
      "مونابينك https://t.co/yyUfIn1OPH\n",
      "Seminario Estructuración, Diseño y Detallado de Acuerdo al Reglamento Estructural Panameño REP 2014, por el Rector… https://t.co/AZQaAJG0RC\n",
      "Me alegra que sigas contándole y juntándote con esa gente que juraste no era nada. #ItsDone 😌\n",
      "*Drunk Joey watching a hot dogs eating contest*\n",
      "\"I've never seen a guy so happy to shove weiners down his throat......\"\n",
      ".....\n",
      "@MADE__USA @Reuters This could be very interesting! Let's see what happens!\n",
      "RT @tdubb59: you're an idiot if you drink and drive point blank. not even just tonight... just don't do it. if you're by me and need a ride…\n",
      "ねむみ\n",
      "اللهم إنك خلقت نفسي وأنت توفاها لك مماتها ومحياها ، إن أحييتها فاحفظها ، وإن أمتها فاغفر لها . اللهم إني أسألك العافية #أذكار_النوم #hadith\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-39-99f3939bf8e7>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     40\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     41\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstream_results\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 42\u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0mline\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mstream_results\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0miter_lines\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     43\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     44\u001b[0m         \u001b[0mdecoded_line\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'utf-8'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/requests/models.py\u001b[0m in \u001b[0;36miter_lines\u001b[0;34m(self, chunk_size, decode_unicode, delimiter)\u001b[0m\n\u001b[1;32m    789\u001b[0m         \u001b[0mpending\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    790\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 791\u001b[0;31m         \u001b[0;32mfor\u001b[0m \u001b[0mchunk\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0miter_content\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchunk_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mchunk_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdecode_unicode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdecode_unicode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    792\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    793\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mpending\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/requests/models.py\u001b[0m in \u001b[0;36mgenerate\u001b[0;34m()\u001b[0m\n\u001b[1;32m    745\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraw\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'stream'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    746\u001b[0m                 \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 747\u001b[0;31m                     \u001b[0;32mfor\u001b[0m \u001b[0mchunk\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraw\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstream\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchunk_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdecode_content\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    748\u001b[0m                         \u001b[0;32myield\u001b[0m \u001b[0mchunk\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    749\u001b[0m                 \u001b[0;32mexcept\u001b[0m \u001b[0mProtocolError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/urllib3/response.py\u001b[0m in \u001b[0;36mstream\u001b[0;34m(self, amt, decode_content)\u001b[0m\n\u001b[1;32m    430\u001b[0m         \"\"\"\n\u001b[1;32m    431\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchunked\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msupports_chunked_reads\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 432\u001b[0;31m             \u001b[0;32mfor\u001b[0m \u001b[0mline\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_chunked\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mamt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdecode_content\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdecode_content\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    433\u001b[0m                 \u001b[0;32myield\u001b[0m \u001b[0mline\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    434\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/urllib3/response.py\u001b[0m in \u001b[0;36mread_chunked\u001b[0;34m(self, amt, decode_content)\u001b[0m\n\u001b[1;32m    592\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_error_catcher\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    593\u001b[0m             \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 594\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_update_chunk_length\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    595\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchunk_left\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    596\u001b[0m                     \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/urllib3/response.py\u001b[0m in \u001b[0;36m_update_chunk_length\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    534\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchunk_left\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    535\u001b[0m             \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 536\u001b[0;31m         \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    537\u001b[0m         \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mb';'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    538\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py\u001b[0m in \u001b[0;36mreadinto\u001b[0;34m(self, b)\u001b[0m\n\u001b[1;32m    584\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    585\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 586\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    587\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    588\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_timeout_occurred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py\u001b[0m in \u001b[0;36mrecv_into\u001b[0;34m(self, buffer, nbytes, flags)\u001b[0m\n\u001b[1;32m   1000\u001b[0m                   \u001b[0;34m\"non-zero flags not allowed in calls to recv_into() on %s\"\u001b[0m \u001b[0;34m%\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1001\u001b[0m                   self.__class__)\n\u001b[0;32m-> 1002\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnbytes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1003\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1004\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0msocket\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnbytes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mflags\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py\u001b[0m in \u001b[0;36mread\u001b[0;34m(self, len, buffer)\u001b[0m\n\u001b[1;32m    863\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Read on closed or unwrapped SSL socket.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    864\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 865\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sslobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    866\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mSSLError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    867\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mSSL_ERROR_EOF\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msuppress_ragged_eofs\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py\u001b[0m in \u001b[0;36mread\u001b[0;34m(self, len, buffer)\u001b[0m\n\u001b[1;32m    623\u001b[0m         \"\"\"\n\u001b[1;32m    624\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mbuffer\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 625\u001b[0;31m             \u001b[0mv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sslobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    626\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    627\u001b[0m             \u001b[0mv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sslobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Import libraries\n",
    "\n",
    "import requests\n",
    "from requests_oauthlib import OAuth1\n",
    "import json\n",
    "from urllib.parse import urlparse\n",
    "\n",
    "#Endpoints\n",
    "url_rest = \"https://api.twitter.com/1.1/search/tweets.json\"\n",
    "url_streaming = \"https://stream.twitter.com/1.1/statuses/sample.json\"\n",
    "\n",
    "#Define parameters\n",
    "params = {\n",
    "'app_key':'YOUR_APP_KEY',\n",
    "'app_secret':'YOUR_APP_SECRET',\n",
    "'oauth_token':'USER_OAUTH_TOKEN',\n",
    "'oauth_token_secret':'USER_OAUTH_TOKEN_SECRET'\n",
    "}\n",
    "\n",
    "auth = OAuth1(\n",
    "    auth_params['app_key'], \n",
    "    auth_params['app_secret'], \n",
    "    auth_params['oauth_token'], \n",
    "    auth_params['oauth_token_secret']\n",
    ")\n",
    "\n",
    "q = 'BMW OR Mercedes OR Audi'\n",
    "\n",
    "params = {'q' : q, 'count' : 100, 'lang' : 'en'}\n",
    "\n",
    "results = requests.get(url_rest,params=params, auth=auth)\n",
    "\n",
    "for tweet in results.json()['statuses']:\n",
    "    print (tweet['text'])\n",
    "    \n",
    "stream_results = requests.get(url_streaming, stream=True,auth=auth)\n",
    "\n",
    "for line in stream_results.iter_lines():\n",
    "    try:\n",
    "        decoded_line = line.decode('utf-8')\n",
    "        print(json.loads(decoded_line)['text'])\n",
    "    except:\n",
    "         pass\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Facebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': [{'message': \"L'image parfaite. \\n#MBPhotoCredit: IG @turbo_temme\\n\\n#MercedesBenz #CLA250 #4MATIC\", 'created_time': '2017-07-04T16:30:00+0000', 'id': '297744124702_10155545975884703'}, {'message': 'Thomas avait perdu de vue son fils, comment rattraper le temps perdu? Découvrez l’histoire. \\n\\n#GrowUp #MercedesBenz #ClasseB', 'created_time': '2017-07-04T09:30:00+0000', 'id': '297744124702_10155533676789703'}, {'message': 'Le retour de la séduction.\\n#MBPhotoCredit: Deyan Yordanov\\n\\n#MercedesAMG #AMG #C43 #Cabriolet', 'created_time': '2017-07-03T16:30:00+0000', 'id': '297744124702_10155545118369703'}, {'message': 'Avez-vous vu notre section Pinterest Mercedes-Benz France?', 'created_time': '2017-07-03T09:30:00+0000', 'id': '297744124702_10155545138114703'}, {'message': 'Mon levrier ❤', 'created_time': '2017-07-02T20:40:29+0000', 'id': '297744124702_10155543243359703'}, {'message': \"Le CLA Shooting Brake: conçu pour exceller.\\n#MercedesBenz #CLA #ShootingBrake \\n\\nPlus d'informations sur: http://benz.me/Sbclafrfb\", 'created_time': '2017-07-02T16:30:00+0000', 'id': '297744124702_10155528670169703'}, {'message': \"Grow up. Dîner en famille. \\n#GrowUp #MercedesBenz #ClasseA\\n\\nPlus d'informations sur la Classe A sur: http://benz.me/ngcc_cafbfr\", 'created_time': '2017-07-02T09:30:00+0000', 'id': '297744124702_10155528676444703'}, {'message': \"Un témoignage pour remercier le garage de l'étoile à Brest. \\nOriginaire de la région parisienne et en vacances dans la région, j'ai été victime d'une panne (platine de la boîte auto) sur ma classe e cabriolet. Arrivé à l'improviste au garage, je suis pris en charge par un personnel compétent à l'écoute. Des billets de tram nous (avec ma compagne) sont gracieusement offert le temps d'inspecter le véhicule. \\nLe véhicule devant être immobilisé, un véhicule nous est proposé (gla quasi neuve). À notre retour le lendemain, nous sommes accueillis par un personnel impliqué qui nous offre une prise en charge sur la facture et des goodies. \\nMerci à toute l'équipe de cette concession qui nous est apparus exemplaire dans la prise en charge de mon véhicule.\", 'created_time': '2017-07-01T22:02:06+0000', 'id': '297744124702_10155539284074703'}, {'message': 'Confortable, performante et stylisée à la perfection. Voici la Mercedes-AMG S 63 4MATIC Cabriolet. \\n#MercedesAMG #AMG #S63 #4MATIC #Cabriolet\\n\\n#MBPhotoCredit: Mercedes-Benz Berlin', 'created_time': '2017-07-01T16:30:00+0000', 'id': '297744124702_10155528649934703'}, {'message': 'Bonjour Chère maison Mercedes-Benz 😉 Je suis né avec la passion de Mercedes-Benz, et je serai toujours fidèle à Mercedes 😍😎, étant aussi un Youngtimer, je vous présente ma Mercedes Classe S W140 la Mercedes dite de \"Lady Di\" 😔 et aussi appelé \"The Phantom\" par son classe et son prestige 😊 la version que j\\'ai est une S350 Turbo-Diesel, uniquement 20518 exemplaires 😮. \\nLa particularité de la mienne, c\\'était un véhicule diplomatique, très bien entretenue, je l\\'ai fait certifier chez Mercedes \"High Mileage Award\" puisque elle a dépassé les 500.000km 🏆🏆🏆, moteur/boite et accessoires d\\'origine! La prochaine médaille est prévue à 750.000km.\\nÉquipée d\\'options atypiques surtout à l\\'époque; comme par exemple la direction assistée à assistance variable (mode route/ville) automatique, la première au monde à être équipée radars de reculs à l\\'ancienne (2 petites antennes qui sortent de l’extrémité du coffre de façon inclinée pour aider le conducteur à l’œil nu de calculer son recul), climatisation automatique Bi-zone, fermeture assistée des portes ect... C\\'était la dernière classe S la plus longue et la plus lourde surtout de sa génération!\\nMercedes avant de la certifier, elle a bien vérifiée que les principaux \"organes\" sont bien d\\'origine et c\\'est bien le cas 😎😍 et j\\'en suis fier 😜 Elle vient de fêter ses 22 ans  🎁🎉🎊 mais elle ne l\\'ai fait pas 👶 \\nJe vous souhaite un très bon week-end  🙂 profitez bien du beau temps 😝☀🌷🍄💐', 'created_time': '2017-07-01T12:47:08+0000', 'id': '297744124702_10155537232439703'}, {'message': 'Toujours regarder vers l’avenir, mais ne jamais oublier le passé. \\nLa légende de la Mercedes SL 73 AMG ne risque pas d’être délaissée.\\n\\n#MercedesAMG #AMG #SL73 #AMG50Ans', 'story': 'Mercedes-Benz France added 4 new photos.', 'created_time': '2017-07-01T09:30:00+0000', 'id': '297744124702_10155528662429703'}, {'message': 'Voici en exclusivité l’intérieur du Mercedes-AMG SLC RedArt Edition.\\n\\n#MercedesAMG #AMG #SLC #RedArtEdition', 'story': 'Mercedes-Benz France added 2 new photos.', 'created_time': '2017-06-30T16:30:00+0000', 'id': '297744124702_10155528616479703'}, {'message': \"Croyez-vous au coup de foudre?\\n#MercedesBenz #CLS #ShootingBrake\\n\\nPlus d'informations sur le CLS Shooting Brake sur: http://benz.me/clssb_fbfr\", 'story': 'Mercedes-Benz France added 3 new photos.', 'created_time': '2017-06-30T09:30:00+0000', 'id': '297744124702_10155528634434703'}, {'message': 'Marc Lavoine, icône de la scène artistique française et ambassadeur Mercedes-Benz, vient de recevoir sa toute Nouvelle Classe E All-Terrain. \\nPour plus d’informations sur ce modèle, rendez-vous sur: http://benz.me/mballtterain_fbfr \\n\\n#MBPhotoCredit: O. Bos', 'story': 'Mercedes-Benz France added 2 new photos.', 'created_time': '2017-06-29T16:30:00+0000', 'id': '297744124702_10155528314779703'}, {'message': \"Vous ne pensez pas qu’il serait temps de grandir? \\n#GrowUp #MercedesBenz #CLACoupé ASAP Rocky\\n\\nPlus d'informations sur: www.grow-up-mercedes.fr\", 'created_time': '2017-06-29T09:30:00+0000', 'id': '297744124702_10155528359164703'}, {'message': 'Dès à présent disponible sur commande, découvrez la nouvelle version Coupé du Mercedes-AMG GLC 63 4MATIC+. \\nL’alliance exclusive de la sportivité Mercedes-AMG et du moteur V8 bi-turbo pour des sensations de conduite inégalées. \\n\\nConfigurez-le ici: http://benz.me/glccoupefb_fr', 'created_time': '2017-06-28T16:30:00+0000', 'id': '297744124702_10155501538864703'}, {'message': 'Bonjour qui peux le renseigner des lettres ATP sur classe c', 'created_time': '2017-06-28T10:45:56+0000', 'id': '297744124702_10155524588779703'}, {'message': \"En ville ou à la campagne, le SL 450 s'adapte parfaitement.\\n\\n#MBPhotoCredit: Jason Fitzzz\\n#MercedesBenz #SL #Roadster\", 'story': 'Mercedes-Benz France added 11 new photos.', 'created_time': '2017-06-28T09:30:00+0000', 'id': '297744124702_10155524174754703'}, {'message': 'Ce qui impressionnait il y a 50 ans est toujours aussi saisissant aujourd’hui. Après de nombreuses rénovations et élargissements, la piste de test top secrète fut présentée aux médias le 9 Mai 1967.\\n\\n#MercedesBenz #Classic', 'story': 'Mercedes-Benz France added 3 new photos.', 'created_time': '2017-06-27T16:30:00+0000', 'id': '297744124702_10155519595244703'}, {'message': \"Il est temps de grandir. Se marier. \\n#GrowUp #MercedesBenz #CLA #ShootingBrake\\n\\nPlus d'informations sur: www.grow-up-mercedes.fr\", 'created_time': '2017-06-27T09:30:00+0000', 'id': '297744124702_10155519615234703'}, {'message': 'Dès maintenant disponible sur commande, découvrez le tout Nouveau Mercedes-AMG GLC 63 4MATIC+. \\nLa combinaison parfaite entre la performance AMG et le moteur V8 bi-turbo. \\n\\nConfigurez-le: http://benz.me/glcsuvfb_fr', 'created_time': '2017-06-26T16:30:00+0000', 'id': '297744124702_10155501536489703'}, {'message': 'Asseyez-vous et profitez de la vue.\\n#MercedesAMG #AMG #G63\\n\\n#MBPhotoCredit: Bas Fransen Photography', 'story': 'Mercedes-Benz France added 3 new photos.', 'created_time': '2017-06-26T09:32:00+0000', 'id': '297744124702_10155515782209703'}, {'message': 'Félicitations à nos pilotes Mercedes-AMG Petronas Motorsport  Valtteri Bottas et Lewis Hamilton pour avoir fini P2 et P5 durant ce week-end de GP en Azerbaijan. #AzerbaijanGP', 'created_time': '2017-06-25T18:15:29+0000', 'id': '297744124702_10155513636964703'}, {'message': 'Pour en savoir plus sur les dernières nouveautés et innovations, rendez-vous sur notre page LinkedIn: https://fr.linkedin.com/company/mercedes-benz-france', 'created_time': '2017-06-25T16:30:00+0000', 'id': '297744124702_10155505406944703'}, {'message': 'Ma', 'created_time': '2017-06-25T13:46:08+0000', 'id': '297744124702_842898749201976'}], 'paging': {'previous': 'https://graph.facebook.com/v2.8/297744124702/feed?since=1499185800&access_token=EAACEdEose0cBAIs7SmmLRPttaZCu6Vhtc2ZApki5aPmFDKIekUchyYOxBFHdtJ2vyFH5nsHbYZChzhRzbrzygDd54siBUPZAfeCoffQge0lX6FqNIabF2AkP6NZB2eaqAYZCS2qfukXzjOZCPTBFRcCoxKESz1fM3abQv34lWrAp0i3YGvrhosq3Mhzb55ZAWKQZD&limit=25&__paging_token=enc_AdBJDEwn56juoGUwBK1O0krieCSFlrzz1VLZCd0q6lji0izy0ZAeTnzyV5pST3e6NVR4dOWPBZByZBQ4u0yxoq2qtbMLjbt7DB6iBcZBGtVbX7QwRYAZDZD&__previous=1', 'next': 'https://graph.facebook.com/v2.8/297744124702/feed?access_token=EAACEdEose0cBAIs7SmmLRPttaZCu6Vhtc2ZApki5aPmFDKIekUchyYOxBFHdtJ2vyFH5nsHbYZChzhRzbrzygDd54siBUPZAfeCoffQge0lX6FqNIabF2AkP6NZB2eaqAYZCS2qfukXzjOZCPTBFRcCoxKESz1fM3abQv34lWrAp0i3YGvrhosq3Mhzb55ZAWKQZD&limit=25&until=1498398368&__paging_token=enc_AdCH559FZBiBpnr4DEaTZB60FQ8OeqZC8grHMArGUSjSZCO9BpRs4rPRRTe3UEtk5gtKsZBFpzutURajVZAozKed2YdHya'}}\n",
      "L'image parfaite. \n",
      "#MBPhotoCredit: IG @turbo_temme\n",
      "\n",
      "#MercedesBenz #CLA250 #4MATIC\n",
      "Thomas avait perdu de vue son fils, comment rattraper le temps perdu? Découvrez l’histoire. \n",
      "\n",
      "#GrowUp #MercedesBenz #ClasseB\n",
      "Le retour de la séduction.\n",
      "#MBPhotoCredit: Deyan Yordanov\n",
      "\n",
      "#MercedesAMG #AMG #C43 #Cabriolet\n",
      "Avez-vous vu notre section Pinterest Mercedes-Benz France?\n",
      "Mon levrier ❤\n",
      "Le CLA Shooting Brake: conçu pour exceller.\n",
      "#MercedesBenz #CLA #ShootingBrake \n",
      "\n",
      "Plus d'informations sur: http://benz.me/Sbclafrfb\n",
      "Grow up. Dîner en famille. \n",
      "#GrowUp #MercedesBenz #ClasseA\n",
      "\n",
      "Plus d'informations sur la Classe A sur: http://benz.me/ngcc_cafbfr\n",
      "Un témoignage pour remercier le garage de l'étoile à Brest. \n",
      "Originaire de la région parisienne et en vacances dans la région, j'ai été victime d'une panne (platine de la boîte auto) sur ma classe e cabriolet. Arrivé à l'improviste au garage, je suis pris en charge par un personnel compétent à l'écoute. Des billets de tram nous (avec ma compagne) sont gracieusement offert le temps d'inspecter le véhicule. \n",
      "Le véhicule devant être immobilisé, un véhicule nous est proposé (gla quasi neuve). À notre retour le lendemain, nous sommes accueillis par un personnel impliqué qui nous offre une prise en charge sur la facture et des goodies. \n",
      "Merci à toute l'équipe de cette concession qui nous est apparus exemplaire dans la prise en charge de mon véhicule.\n",
      "Confortable, performante et stylisée à la perfection. Voici la Mercedes-AMG S 63 4MATIC Cabriolet. \n",
      "#MercedesAMG #AMG #S63 #4MATIC #Cabriolet\n",
      "\n",
      "#MBPhotoCredit: Mercedes-Benz Berlin\n",
      "Bonjour Chère maison Mercedes-Benz 😉 Je suis né avec la passion de Mercedes-Benz, et je serai toujours fidèle à Mercedes 😍😎, étant aussi un Youngtimer, je vous présente ma Mercedes Classe S W140 la Mercedes dite de \"Lady Di\" 😔 et aussi appelé \"The Phantom\" par son classe et son prestige 😊 la version que j'ai est une S350 Turbo-Diesel, uniquement 20518 exemplaires 😮. \n",
      "La particularité de la mienne, c'était un véhicule diplomatique, très bien entretenue, je l'ai fait certifier chez Mercedes \"High Mileage Award\" puisque elle a dépassé les 500.000km 🏆🏆🏆, moteur/boite et accessoires d'origine! La prochaine médaille est prévue à 750.000km.\n",
      "Équipée d'options atypiques surtout à l'époque; comme par exemple la direction assistée à assistance variable (mode route/ville) automatique, la première au monde à être équipée radars de reculs à l'ancienne (2 petites antennes qui sortent de l’extrémité du coffre de façon inclinée pour aider le conducteur à l’œil nu de calculer son recul), climatisation automatique Bi-zone, fermeture assistée des portes ect... C'était la dernière classe S la plus longue et la plus lourde surtout de sa génération!\n",
      "Mercedes avant de la certifier, elle a bien vérifiée que les principaux \"organes\" sont bien d'origine et c'est bien le cas 😎😍 et j'en suis fier 😜 Elle vient de fêter ses 22 ans  🎁🎉🎊 mais elle ne l'ai fait pas 👶 \n",
      "Je vous souhaite un très bon week-end  🙂 profitez bien du beau temps 😝☀🌷🍄💐\n",
      "Toujours regarder vers l’avenir, mais ne jamais oublier le passé. \n",
      "La légende de la Mercedes SL 73 AMG ne risque pas d’être délaissée.\n",
      "\n",
      "#MercedesAMG #AMG #SL73 #AMG50Ans\n",
      "Voici en exclusivité l’intérieur du Mercedes-AMG SLC RedArt Edition.\n",
      "\n",
      "#MercedesAMG #AMG #SLC #RedArtEdition\n",
      "Croyez-vous au coup de foudre?\n",
      "#MercedesBenz #CLS #ShootingBrake\n",
      "\n",
      "Plus d'informations sur le CLS Shooting Brake sur: http://benz.me/clssb_fbfr\n",
      "Marc Lavoine, icône de la scène artistique française et ambassadeur Mercedes-Benz, vient de recevoir sa toute Nouvelle Classe E All-Terrain. \n",
      "Pour plus d’informations sur ce modèle, rendez-vous sur: http://benz.me/mballtterain_fbfr \n",
      "\n",
      "#MBPhotoCredit: O. Bos\n",
      "Vous ne pensez pas qu’il serait temps de grandir? \n",
      "#GrowUp #MercedesBenz #CLACoupé ASAP Rocky\n",
      "\n",
      "Plus d'informations sur: www.grow-up-mercedes.fr\n",
      "Dès à présent disponible sur commande, découvrez la nouvelle version Coupé du Mercedes-AMG GLC 63 4MATIC+. \n",
      "L’alliance exclusive de la sportivité Mercedes-AMG et du moteur V8 bi-turbo pour des sensations de conduite inégalées. \n",
      "\n",
      "Configurez-le ici: http://benz.me/glccoupefb_fr\n",
      "Bonjour qui peux le renseigner des lettres ATP sur classe c\n",
      "En ville ou à la campagne, le SL 450 s'adapte parfaitement.\n",
      "\n",
      "#MBPhotoCredit: Jason Fitzzz\n",
      "#MercedesBenz #SL #Roadster\n",
      "Ce qui impressionnait il y a 50 ans est toujours aussi saisissant aujourd’hui. Après de nombreuses rénovations et élargissements, la piste de test top secrète fut présentée aux médias le 9 Mai 1967.\n",
      "\n",
      "#MercedesBenz #Classic\n",
      "Il est temps de grandir. Se marier. \n",
      "#GrowUp #MercedesBenz #CLA #ShootingBrake\n",
      "\n",
      "Plus d'informations sur: www.grow-up-mercedes.fr\n",
      "Dès maintenant disponible sur commande, découvrez le tout Nouveau Mercedes-AMG GLC 63 4MATIC+. \n",
      "La combinaison parfaite entre la performance AMG et le moteur V8 bi-turbo. \n",
      "\n",
      "Configurez-le: http://benz.me/glcsuvfb_fr\n",
      "Asseyez-vous et profitez de la vue.\n",
      "#MercedesAMG #AMG #G63\n",
      "\n",
      "#MBPhotoCredit: Bas Fransen Photography\n",
      "Félicitations à nos pilotes Mercedes-AMG Petronas Motorsport  Valtteri Bottas et Lewis Hamilton pour avoir fini P2 et P5 durant ce week-end de GP en Azerbaijan. #AzerbaijanGP\n",
      "Pour en savoir plus sur les dernières nouveautés et innovations, rendez-vous sur notre page LinkedIn: https://fr.linkedin.com/company/mercedes-benz-france\n",
      "Ma\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "url = 'https://graph.facebook.com/v2.8/'\n",
    "\n",
    "page_url = 'https://graph.facebook.com/v2.8/{page-id}/feed'\n",
    "page_url = 'https://graph.facebook.com/v2.8/MercedesBenzFrance/feed'\n",
    "page_url = 'https://graph.facebook.com/v2.8/{page-id}/photos'\n",
    "\n",
    "params = {'access_token': 'YOUR_ACCESS_TOKEN'}\n",
    "          \n",
    "\n",
    "page_url = 'https://graph.facebook.com/v2.8/MercedesBenzFrance/feed'\n",
    "\n",
    "result = requests.get(page_url, params = params)\n",
    "          \n",
    "data = result.json()\n",
    "print(data)\n",
    "for element in data['data']:\n",
    "    print(element['message'])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Github"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "API response: {\"message\":\"Bad credentials\",\"documentation_url\":\"https://developer.github.com/v3\"}\n"
     ]
    },
    {
     "ename": "KeyError",
     "evalue": "'token'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-53-a1f88dd804be>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     18\u001b[0m \u001b[0mprint\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m\"API response:\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m \u001b[0mtoken\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjson\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'token'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     21\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     22\u001b[0m \u001b[0murl\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"https://api.github.com/search/repositories\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'token'"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "username = 'YOUR_GITHUB_USERNAME'\n",
    "password = 'YOUR_GITHUB_PASSWORD'\n",
    "\n",
    "url = 'https://api.github.com/authorizations'\n",
    "note = 'my test application'\n",
    "\n",
    "post_data = {'scopes':['user','repo'],'note': note }\n",
    "\n",
    "response = requests.post(\n",
    "    url,\n",
    "    auth = (username, password),\n",
    "    data = json.dumps(post_data),\n",
    "    )   \n",
    "\n",
    "print (\"API response:\", response.text)\n",
    "\n",
    "token = response.json()['token']\n",
    "\n",
    "url = \"https://api.github.com/search/repositories\"\n",
    "url = \"https://api.github.com/search/repositories/q=QUERY+language:LANG+created:>2016-01-01+pushed:>2016-01-01&sort=stars&order=desc\"\n",
    "\n",
    "ACCESS_TOKEN = 'YOUR_ACCESS_TOKEN'\n",
    "access_token_str = '&access_token=' + ACCESS_TOKEN\n",
    "\n",
    "response = requests.get(url + access_token_str)\n",
    "\n",
    "result = response.json()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## YouTube"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'errors': [{'domain': 'usageLimits', 'reason': 'dailyLimitExceededUnreg', 'message': 'Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup.', 'extendedHelp': 'https://code.google.com/apis/console'}], 'code': 403, 'message': 'Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup.'}\n"
     ]
    },
    {
     "ename": "KeyError",
     "evalue": "'items'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-65-a3fe75d48b9a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     36\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'error'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     37\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 38\u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0mr\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'items'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     39\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mr\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'snippet'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'title'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     40\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'items'"
     ]
    }
   ],
   "source": [
    "from oauth2client.client import OAuth2WebServerFlow\n",
    "\n",
    "\n",
    "####Get api_key\n",
    "client_id = 'YOUR_CLIENT_ID'\n",
    "client_secret = 'YOUR_CLIENT_SECRET'\n",
    "\n",
    "flow = OAuth2WebServerFlow(client_id=client_id,\n",
    "                           client_secret=client_secret,\n",
    "                           scope='https://gdata.youtube.com',\n",
    "                           redirect_uri='YOUR_APP_URL')\n",
    "\n",
    "auth_uri = flow.step1_get_authorize_url()\n",
    "\n",
    "\n",
    "# Define URLs\n",
    "url = 'https://www.googleapis.com/youtube/v3/search'\n",
    "url = 'https://www.googleapis.com/youtube/v3/commentThreads'\n",
    "\n",
    "\n",
    "###Get the data\n",
    "import requests, json\n",
    "\n",
    "access_token = api_key\n",
    "\n",
    "#access_token = \"YOUR_ACCESSS_TOKEN\"\n",
    "\n",
    "url = 'https://www.googleapis.com/youtube/v3/search'\n",
    "\n",
    "query = 'ps4'\n",
    "\n",
    "params = {'part': 'snippet', 'q': query, 'type': 'video', 'videoCaption': 'closedCaption', 'access_token': access_token}\n",
    "\n",
    "response = requests.get(url, params = params)\n",
    "\n",
    "data = response.json()\n",
    "\n",
    "for r in data['items']:\n",
    "    print(r['snippet']['title'])\n",
    "    \n",
    "video_id = r['id']['videoId']\n",
    "\n",
    "\n",
    "url = 'https://www.googleapis.com/youtube/v3/commentThreads'\n",
    "params = {'part': 'snippet', 'videoId': video_id  , 'access_token': access_token}\n",
    "\n",
    "response = requests.get(url, params = params)\n",
    "\n",
    "data = response.json()\n",
    "\n",
    "for r in data['items']:\n",
    "   print(r['snippet']['topLevelComment']['snippet']['textDisplay'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pinterest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': {'url': 'https://www.pinterest.com/appleprog/', 'first_name': 'James', 'last_name': 'Bond', 'id': '434034620246293422'}}\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "access_token = 'YOUR_ACCESS_TOKEN'\n",
    "\n",
    "params = {'access_token': access_token, 'fields' : 'counts, bio, first_name, last_name'}\n",
    "\n",
    "\n",
    "def pinterest(url, params):\n",
    "\n",
    "    response = requests.get(url = url, params = params)\n",
    "\n",
    "    return response.json()\n",
    "\n",
    "me = pinterest(url = 'https://api.pinterest.com/v1/me', params = {'access_token': access_token})\n",
    "\n",
    "followers = pinterest(url = 'https://api.pinterest.com/v1/me/followers/', params = {'access_token': access_token, 'fields': 'counts, username'})['data']\n",
    "\n",
    "following = pinterest(url = 'https://api.pinterest.com/v1/me/following/users/', params = {'access_token': access_token, 'fields': 'counts, username'})['data']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic cleaning techniques"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Remove whitespaces\n",
      "\n",
      "So coooooooool ! I just love thisGreat product, <div>Great Product</div> http://www.greatproduct.com !!!\n",
      "\n",
      "Remove html tags\n",
      "\n",
      "So coooooooool ! I just love thisGreat product,  Great Product  http://www.greatproduct.com !!!\n",
      "\n",
      "Remove urls\n",
      "\n",
      "So coooooooool ! I just love thisGreat product,  Great Product   \n",
      "\n",
      "Remove ponctuation\n",
      "\n",
      "So coooooooool  I just love thisGreat product  Great Product   \n",
      "\n",
      "Standardize words\n",
      "\n",
      "So cool  I just love thisGreat product  Great Product  \n",
      "\n",
      "Split attached words\n",
      "\n",
      "So cool   I just love this Great product   Great  Product  \n",
      "\n",
      "Lowercase\n",
      "\n",
      "so cool   i just love this great product   great  product  \n",
      "\n",
      "Stopwords\n",
      "\n",
      "cool love great product great product\n",
      "\n",
      "Tokenize\n",
      "\n",
      "['cool', 'love', 'great', 'product', 'great', 'product']\n"
     ]
    }
   ],
   "source": [
    "import re, itertools\n",
    "import nltk\n",
    "from nltk.corpus import stopwords \n",
    "\n",
    "verbatim = '    So coooooooool ! I just love thisGreat product, <div>Great Product</div> http://www.greatproduct.com !!!'\n",
    "\n",
    "print(\"Remove whitespaces\\n\")\n",
    "verbatim = verbatim.strip()\n",
    "print(verbatim)\n",
    "\n",
    "print(\"\\nRemove html tags\\n\")\n",
    "verbatim = re.sub(r'<[^<]+?>', ' ', verbatim)\n",
    "print(verbatim)\n",
    "\n",
    "print(\"\\nRemove urls\\n\")\n",
    "verbatim = re.sub(r'https?:\\/\\/.*[\\r\\n]*', ' ', verbatim, flags=re.MULTILINE)\n",
    "print(verbatim)\n",
    "\n",
    "print(\"\\nRemove ponctuation\\n\")\n",
    "verbatim = re.sub(r'[^\\w\\s]','',verbatim)\n",
    "print(verbatim)\n",
    "\n",
    "print(\"\\nStandardize words\\n\")\n",
    "verbatim = ''.join(''.join(s)[:2] for _, s in itertools.groupby(verbatim))\n",
    "print(verbatim)\n",
    "\n",
    "print(\"\\nSplit attached words\\n\")\n",
    "verbatim = ' '.join(re.findall('[A-Z][^A-Z]*', verbatim))\n",
    "print(verbatim)\n",
    "\n",
    "print(\"\\nLowercase\\n\")\n",
    "verbatim = verbatim.lower()\n",
    "print(verbatim)\n",
    "\n",
    "print(\"\\nStopwords\\n\")\n",
    "verbatim = ' '.join([word for word in verbatim.split() if word not in (stopwords.words('english'))])\n",
    "print(verbatim)\n",
    "\n",
    "print(\"\\nTokenize\\n\")\n",
    "tokens = nltk.word_tokenize(verbatim)\n",
    "print(tokens)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n",
      "2\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "example = [('line1','value1'),('line2','value1'),('line3','value')]\n",
    "df = pd.DataFrame(example)\n",
    "df.columns = ['line','value']\n",
    "\n",
    "\n",
    "#df = df.drop_duplicates(subset=['colname'])\n",
    "\n",
    "print(len(df))\n",
    "df = df.drop_duplicates(subset=['value'])\n",
    "print(len(df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mongo DB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
